Building Collections with ccBuild

ccBuild is one of the three utilities (Library Manager, and Manage Content being the other two) you can to build and update a content collection. The ccBuild utility is included in the NXT Builder software. Building a content collection with ccBuild is different than building a collection with Library Manager or Manage Content. ccBuild is a command-line utility that enables you to build a single content collection from files located on your file system. ccBuild, like Library Manager and Manage Content, produces a completely indexed, transportable, self-contained, discrete, and secure file containing a collection of documents in a hierarchical structure. And, all documents that reside in the content collection reside in their native format and are browse-able, searchable, and retrievable from your NXT site.

The process of using ccBuild to create a content collection is significantly different than with using the Library Manager or Manage Content. Like the Library Manager, ccBuild can create a content collection that contains potentially thousands of documents with very little effort, albeit one at a time. However, the ccBuild process offers you a degree of control more than Library Manager over how a content collection is built, structured, and indexed. The ccBuild process contains many parts necessary for ccBuild to do its work. ccBuild is but one part in that process. The following is a list of required and optional pieces you need to create a content collection with ccBuild:

• makefile (required)
• Document Type Definition (DTD) file (required)
• MakeStart Utility (optional)
• ccBuild Utility (required)
• Index Sheets (optional)
• content collection - NXT file (result of build)

This Documentation addresses the elements and attributes of a content collection makefile. If you are unfamiliar with XML, please consult available XML documentation as this Documentation is not a XML primer. This Documentation discusses XML in general terms and covers the specifics of the makefile.dtd for content collections.

The makefile, DTD file, and ccBuild are equally dependent on each other's existence to create a content collection. If any one of these pieces are missing, you are not able to create a content collection with ccBuild. Figure 1 is a graphical representation of the association and integration of these files (excluding the SDF) in creating a content collection.

Figure 1. The ccBuild Process

Makefile

The content collection makefile is the key piece that enables ccBuild to build a content collection. A makefile is an XML document that contains instructions ccBuild follows to build a content collection. There is one-to-one correspondence between a content collection and a makefile; meaning, for every content collection there is a specific makefile, and vice-versa. A makefile contains relevant and necessary information regarding the content collection in general, and lists information respective to each document that ccBuild will include in the content collection. That information includes:

• Location of source content
• Structure of table of contents
• Display settings of content
• Content Handling
• Content Identification
• Other

Because these issues are dealt with within the makefile, you have total control over nearly every aspect of your content. The makefile must conform to the content collection makefile DTD and be UTF-8 encoded.

UTF-8 encoding is assumed and does not need to be specified in the makefile, although depending on the editor used to create the makefile, you may need to select UTF-8 when saving the file (this is the case with Notepad and other text editors). The name and location of your makefile is up to you. Makefiles generally use the extension .mak, however, this is merely for convenience in differentiating a makefile from other XML documents. Feel free to use other extensions that do not cause conflicts or confusion with other applications (for example, your_makefile.bob would also work).

Original Content Location

The makefile contains a list of all the files for ccBuild to include in your content collection. Each file in this list is a XML "document" element. Each document element contains many attributes. One document-element attributes is the "location" attribute that refers, or points to the actual document on your file system. ccBuild uses this location attribute to locate and retrieve the respective document. The following is an example of a document element that you find in a makefile:

<document title="ccBuild" 
	name="ccbuild.htm" 
	location="c:\<path_to_document>\ccbuild.htm" 
	content-type="text/html" 
	indexsheet="def-indexsheet"/>

If the document has been moved or does not exist in the location specified by the makefile, ccBuild will give you an error indicating that the specified document cannot be found in the specified location. This error is not a fatal error and will not terminate the build process. ccBuild will simply issue the error and proceed to the next document in the makefile. If the location attribute of a document element is not specified in the makefile, ccBuild treats this as a document without associated content and will show up as an empty folder in your content collection.

You can have a content collection that contains documents originating from different locations, just so long as the "location" attribute is pointing to the appropriate location for that file. However, after ccBuild builds the content collection, the original location becomes irrelevant, because all of the content, relevant to the content collection, then resides within the content collection, or NXT file. The original location of the content will not be used again until ccBuild is again executed to update the content collection.

Table of Contents Structure

The makefile controls the structure of the table of contents for a content collection. (See Figure 2) This means you control the hierarchical structure of your content collection, because you are able to control the makefile. You can manipulate the structure of your content collection (and ultimately what you see on your site) by arranging and nesting the XML document elements in your makefile. Because the makefile is an XML document, you must follow proper XML tagging and nesting rules.

Document elements that have other document elements nested within them appear in the table of contents on your site as click-able folder icons , with sub-documents or folders inside. Document elements without document elements nested within them appear in the table of contents on your NXT site as document icons . Figure 2 shows the nesting of document elements within the makefile.

Figure 2. Site vs. makefile Structure

Content Display

The makefile determines whether documents are visible in your content collection. You can designate whether a file appears in the table of contents or not by adding the hidden="yes" attribute and value to a document element of the makefile . You cannot, however with the makefile, request that only part of a document be hidden. (That is a job for the Index sheet.)

You can hide a single document or a folder. Hiding a document hides that document only, however, hiding a folder (a document element with nested elements) hides the folder and also the documents within that folder. Images and style sheets are common examples of files that you may typically hide rather than display in the table of contents.

Content Handling

The makefile also directs the processing of the content going into the content collection with ccBuild. The makefile does not do the actual content processing, but holds the processing instructions for each document. Think of ccBuild as the General Contractor and the makefile is the set of blueprints for building to build. The blueprint shows what needs to be done and instructions to accomplish the work. The General Contractor may do some of the work himself or will contract it out to sub-Contractors. Similarly, the makefile indicates the work to be done and "who" is to do the work. ccBuild either does the task itself or contracts the work out. The makefile contains instructions for ccBuild on what needs to be done to each document ccBuild retrieves.

There are many content handling options that you can designate through document element attributes in your makefile; content collection encryption, character encoding, data compression, or special indexing (Index sheets). Some of these options ccBuild accomplishes and others are "contracted out" to others. However, the "document handler" (or sub-Contractor) that is first to "handle" the content is the Document Source Extension.

Document Source Extensions

Document Source Extensions (DSE) are modules that ccBuild uses to read, retrieve, and import source documents that ccBuild stores in a content collection. A DSE acts as a "middle-man" that "sits" between ccBuild and the document storage system (file system, web site, document management system, etc.). DSEs are responsible for reading source documents from the document's native location and delivering that data to ccBuild. (Kind of like a delivery service for the General Contractor.) DSEs can also, optionally, preprocess documents. Meaning, DSEs can add, change, or remove data to, within, or from a given document. A DSE also has the ability to change the format of any or all documents it delivers to ccBuild. The DSE preprocesses the documents before sending them to ccBuild to place in the content collection.

ccBuild uses the File System DSE (fsysdse) to import graphics, HTML, Word, Excel, PowerPoint, ODT, PDF, and XML documents from your file system. This DSE module does not pre-process the documents before sending them to ccBuild. The document remains in its native format. Therefore, within the makefile, the "content-type" document element attribute should correspond to the data format of the source document. Otherwise, the content-type attribute should correspond to the data format of the resultant DSE-processed document. DSE architecture, in general, supports "chaining". Chaining is the process of one DSE receiving the output provided by another DSE. However, fsysdse does not support chaining.

The content type of a document determines whether or not the document is indexed. If the content type allows for indexing NXT indexes that document according to the language format of the document.

Note: To index ODT documents, you must install the corresponding version of OpenOffice IFilters that is included in the Apache OpenOffice 4.0 and higher, or similar software (for example, LibreOffice 4.3 and higher). You can download and install the required software manually from the official site.

Language

Another facet of content handling is designating the language to use in creating your content collection. You determine and apply the appropriate language of your content on the content collection level not the individual document level. The language attribute of the content collection element determines how NXT parses and indexes the text within the content collection. The default language module is English (US). NXT 4 supports the following language modules: English (US), English (UK), Spanish, French, German, Dutch, Portuguese, and Japanese. (Japanese is available as an add-on module.)

Other

There are other issues that you can control through the makefile. You can specify a password for your content collection, which would be required to add the content collection to any site, and very relevant if you employ License Management. You can designate a name and location for a file to serve as an update file for your content collection. There are many other options you can designate through the makefile. All of these options are governed by the rules found in the Document Type Definition file.

Document Type Definition File

A Document Type Definition (DTD) file is a document that contains rules that govern the structure of an XML document. These rules consist of a list of allowable XML elements and their respective attributes. Not all XML files have DTDs to govern their structure and existence. However, there is a DTD that does govern the structure of the NXT makefile. NXT makefiles conform to the makefile.dtd DTD. The makefile.dtd is specifically designed to work with the NXT makefiles. DTDs contain rules and guidelines for the following three items:

• Entities
• Elements
• Attributes

Entities

A DTD entity is like a global constant or alias. DTD entities allow you to represent a long string of text with a short string. DTD entities also define default values DTD elements can use. DTD entities in the makefile.dtd are listed toward the top of the file. Figure 3 shows some of the entity declarations of the makefile.dtd.

Figure 3.  Makefile.dtd   Entity Declarations

Should you need to change these Entities, do not change them by modifying makefile.dtd. Instead, you can override the makefile.dtd Entities by redefining them in the makefile's internal DTD subsection. (See makefile for example.)

Elements

The term element is the same as in XML and HTML. Elements within a DTD define the "elements" (start and end tags) permissible in an XML document. Any element within that XML document, must be defined within the associated DTD. A DTD element definition may include possible sub-elements that you can nest within a given element. A DTD lists, in parentheses, the sub-elements following the element declaration <!ELEMENT and element name. Figure 4 shows some examples of element declarations.

Figure 4.  Makefile.dtd   Element Declarations

The order in which the DTD lists the sub-elements (comma delimited) in the element declaration is the nesting order they must appear within the element of the makefile. Each has The symbol following each sub-element designates the quantity of sub-elements that are allowable within a given element. Table 1 lists the quantity symbols and their respective quantitative meanings.

Table 1.  Makefile.dtd   Symbol Definitions

All elements and sub-elements must have an element declaration and definition within the DTD. Generally, the DTD defines sub-elements after the element that references them, however order does not matter. An element definition also consists of a list of element-relevant attributes allowable for the given element.

Attributes

Generally, the attributes of an element comprise the bulk of a given element definition. In an XML document, an element's start tag contains the element's attributes. Depending upon the rules in the DTD, some attributes are required and some are not. Figure 5 shows the document element declaration with its subsequent attributes list from the makefile.dtd, and a document element from the makefile with a subset of the applicable attributes. Notice that some attributes in the DTD section are indicated as "required", some as "implied", which means not required, and some with set default values.

Figure 5.  Makefile.dtd   Element Attributes

Each element in the makefile.dtd has its own set of attributes. You can add, edit, and delete nearly any of an element's attributes, except for the required ones.

MakeStart Utility

The MakeStart utility is a Windows-interface utility included in the NXT Builder software, and assists you in writing or creating makefiles. You do not have to use this utility to create a makefile. Instead, you can choose to create a makefile manually with an HTML, XML, or text editor. You will save valuable time and frustration (from debugging) if you use the MakeStart utility, if for nothing else, than starting your makefile.

Launch the MakeStart utility by choosing Start..Programs..Rocket..NXT 4..Builder..Make File Starter. Figure 6 shows the MakeStart utility user interface and Table 1 a list of the information needed to create your makefile.

Figure 6. MakeStart User Interface

For MakeStart to create your makefile, you need to provide information, or "bits" of data, through this interface. The MakeStart interface is organized into three information categories, with each category containing multiple input fields. The information that you type in these fields enable MakeStart to construct a makefile. The three information categories are:

• Makefile information
• Content Collection information
• Documents information

Makefile Information

For MakeStart to create your makefile, you need to provide the following information for naming, locating, and building your makefile:

• File name is the full path (name and location path) of the makefile that MakeStart will create. You can browse to a location for your makefile by clicking the button. After selecting the directory, type the name of your makefile in the File name field of the browse window.
• DTD file name is the full path (name and location path) of the makefile.dtd that governs the creation of the makefile. MakeStart populates this field with the following default path: [application folder]\bin\makefile.dtd.

Content Collection Information

Ultimately, the ccBuild process produces an NXT content collection. Therefore, since ccBuild ultimately creates a content collection, and since ccBuild only acts on the set of instructions in a makefile, then the makefile must have information for the creation and identification of the content collection within it. And, since MakeStart builds makefiles, then it makes sense that you need to provide information about your content collection to the MakeStart utility. In fact, you need to provide the following required information pertaining to your eventual content collection:

• File name is the full path (name and location path) of the content collection (NXT file) that ccBuild will create. You can browse to a location for the NXT file by clicking the button. After selecting the directory, type the name of your NXT file in the File name field of the browse window with the .nxt extension. NXT 4 supports content collections with the .nxt extension.
• ID is the site-wide unique identifier of your content collection that ccBuild creates.
• Title is the text that shows up in your table of contents on your site.

Documents Information

The makefile and content collection information basically creates the NXT file container, but does not "fill" that container with any content. The documents information is where "the rubber meets the road" and you designate the content you want in the content collection and how to handle the indexing. The following information has a great impact on the contents of your content collection, although not all of the following information is required:

• Directory to traverse is required. This field is where you direct MakeStart and ultimately ccBuild (and the DSE) where the content on your file system resides that you want in your content collection. You can browse to the appropriate directory location by clicking the button.
• Traverse sub directories check box enables you to control whether ccBuild includes content only in the folder of the directory that you indicate or includes the content in that folder and all sub folders within that directory.
• Include/Exclude files with these extensions options and data field enable you to designate by file type the content you want to include or not by typing the file extensions (semi-colon delimited) in the field. The default is to include all files within the indicated directory.
• Hide files with these extensions enables you to include files in your content collection but not display them. For files that have extensions in this field, MakeStart will create document elements in the makefile with the hidden="yes" attribute.
• Index sheet for HTML/XML enables you to identify an index sheet to index your HTML and XML documents. This field only allows for one index sheet reference. You can browse to the appropriate index sheet by clicking the button. Remember that NXT 4, without index sheets, removes tags (breaks are not inserted) and performs full-text indexing of your content. Index sheets enhance that index by providing custom indexing capabilities.

MakeStart Limitations

There are many features available through the makefile which are not accessible through MakeStart. MakeStart sets only minimal requirements for building a content collection and are generally acceptable for a prototype. Also, because MakeStart uses the source directory structure, you are restricted to the way the file system orders the folders and individual files within each folder.

If you want more control of your content collection build process, you can edit the makefile produced by MakeStart and make some of the following additions.

• Include one or more index sheets such as the default index sheet for HTML documents (HTML.XIL). Set one index sheet as the default or set the index sheet attribute for each document. Failing to set up index sheets reports lots of warnings when the content collection is built.
• Determine which documents should be hidden from the table of contents. For example, you might want to hide image files and style sheets. A search and replace on the makefile is an easy way to change multiple documents (replacing content-type with content-type plus the hidden attribute).
• Look for documents without a content type. Any document that has the location attribute set should have a content type. ccBuild reports warnings on these documents when building.
• Run ccBuild in verify mode (the /V option) so you can detect most, but not all, remaining problems that might exist.

Regarding the location of your documents, the MakeStart utility creates a makefile with references, or pointers, to the documents located in a file system directory. You can manually add any documents that are outside the main directory, which you want in your content collection after you run the MakeStart utility and create your makefile for the first time.

You may create your makefile manually without the assistance of MakeStart. However, you must include information listed for the MakeStart utility, as well as other necessary information as dictated by the makefile.dtd. Proper placement of this information within the makefile is as important as having the information there at all. ccBuild will not build your content collection if the makefile is not completely valid. On the other hand, MakeStart can do all of this automatically for you. MakeStart is a utility for creating not editing makefiles.

ccBuild Utility

ccBuild is a command line utility you may use to create and build a content collection. ccBuild creates and builds a content collection by executing instructions in a makefile. However, ccBuild, first, validates the structure and content of the makefile against the makefile.dtd before it executes any makefile instructions and processes any documents. If the makefile fails the validation, ccBuild will report the error and will abort the build process. For more information on the makefile and DTD see subsequent topics in this section. Figure 7 shows the command-line usage for ccBuild

Figure 7. ccBuild Usage Summary

For more information on ccBuild command-line syntax and usage see ccBuild documentation under Publishing Tools.

Index Sheets

An Index sheet, in short, is a document that contains special indexing instructions for XML and HTML documents within your content collection. ccBuild does not require you to use index sheets. You implement index sheets to index your content over and above simple full-text indexing, thus enabling you to search your content by methods other than simple word searches.

The relationship of index sheets to documents within a content collection is a one-to-many. Meaning, one index sheet can index all the documents in your content collection, or you can create a custom index sheet to index each separate document in your content collection. However, only one index sheet can index a single document.

Index sheets are fixed. If you edit them, you must start your collection over (in Library Manager, see Start Collection Over menu item) and will need to redistribute your collection.

The language that you set in your makefile governs both full-text and index sheet-based indexing for your content collection. Makefile.dtd defaults the "lang-module" attribute of the content collection element as "FAST US English Server Extension Module. Version 2.01".

When you implement indexsheets into the ccBuild process, ccBuild passes the documents and files you designate to be indexed by an indexsheet to that respective index sheet for indexing. Once the indexsheet is finished indexing that document, the document and index information is passed back to ccBuild who places the information into your content collection. After all content is indexed and placed and placed in the content collection, ccBuild also places a complete copy of your indexsheets into the content collections as well. ccBuild doing this enables you to "reverse engineer" a content collection. Meaning, you can produce source documents, indexsheets, and a makefile from the content in a content collection.

Content Collection (NXT file)

When ccBuild executes error-free, it builds a content collection. This content collection is an NXT file that contains the content index and table of contents information, and contains a copy of your source the content. This file is the result of a successful build and is deployable on any NXT system.

Summary

When you build a content collection with ccBuild, you need to remember and ensure that all the parts are present and accounted for. So, let's do a quick summary of the most import aspects of this process.

First, and foremost, in order to build a content collection using ccBuild you absolutely need to have:

• A makefile
• makefile.dtd
• ccBuild

Without any one of these three pieces you cannot build a content collection with ccBuild. (Included with ccBuild are a host of DLLs that are not mentioned here, they too are necessary. However, from your perspective, you use ccBuild and ccBuild uses the DLLs to get the job done. You actually use these DLLs indirectly, not directly.) Each is equally dependent on the other. The makefile.dtd tells the makefile what can be done, the makefile tells ccBuild what to do, and ccBuild does it. Of course the direct result of having those three pieces when you run ccBuild is a complete and fully portable content collection, or NXT file.

Second, the NXT file for a content collection contains your content index, table of contents information, and a copy of your source content. Also, although the makefile and ccBuild use your source content to build your content collection, once you execute ccBuild and build your content collection, your content becomes local; a part of the NXT file. This is the content you see on your site.

Third, the MakeStart utility and Indexsheets are optional. You can choose not to use either and still successfully build a content collection with ccBuild. The MakeStart utility is a tool to help you quickly and accurately create or start a makefile. Indexsheets are to provide you with special methods for indexing, and ultimately searching, your content, beyond full-text indexing (and simple text searching). NXT automatically does a full-text index without an Indexsheet.

Fourth and final, using ccBuild, you can create and build only one content collection at a time. And, you can only build a content collection from file system content. To build content collections from other content sources, or to build multiple content collections, refer to the Library Manager application and documentation.

Copyright © 2006-2023, Rocket Software, Inc. All rights reserved.